Validation of scientific topic models using graph analysis and corpus metadata

Abstract Probabilistic topic modeling algorithms like Latent Dirichlet Allocation (LDA) have become powerful tools for the analysis of large collections documents (such as papers, projects, or funding applications) in science, technology an innovation (STI) policy design and monitoring. However, selecting appropriate stable model a specific application (by adjusting hyperparameters algorithm) is not trivial problem. Common validation metrics coherence perplexity, which are focused on quality topics, good fit applications where document similarity relations inferred from especially relevant. Relying graph techniques, aim our work to state new methodology selection specifically oriented optimize emanating model. In order do this, we propose two metrics: first measures variability graphs that result different runs algorithm fixed value hyperparameters, while second metric alignment between derived LDA another obtained using metadata available corresponding corpus. Through experiments various corpora related STI, it shown proposed provide relevant indicators select number topics build persistent models consistent with metadata. Their use, can be extended other beyond LDA, could facilitate systematic adoption this kind techniques STI design.